In general, an image pickup apparatus, such as a digital camera or a video camera, reads image data from an image sensor, such as a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) sensor, and executes various image processing including gamma correction, pixel interpolation, matrix transform, and the like. An image sensor constituted by one plate includes color filters for a plurality of colors. In an image sensor like this, one color filter for any of the plurality of colors is provided to each pixel. As illustrated in FIG. 4, an image sensor having the Bayer array, which includes color filters for the primary colors of red (R), green (G), and blue (B) (RGB), generally has the following recursive pixel pattern. More specifically, in the recursive pixel pattern, one 2 vertical*2 horizontal pixel block of four pixels includes one R pixel and one B pixel, which are diagonally provided, and the other two G pixels, which are reversely diagonally provided.
Image data read from an image sensor may generally include noise components, which may arise on a constituent circuit of the image sensor. Further, a phenomenon of false color may occur in image data read from an image sensor. Accordingly, image data is subjected to image processing for reducing the above-described unnecessary signal components.
For example, there is a conventional method that reduces unnecessary signal components included in image data by separating input image data into a plurality of pieces of image data of different frequency bands and by combining the plurality of pieces of image data of different frequency bands together later.
A method discussed in Japanese Patent Application Laid-Open No. 2008-015741 first hierarchically executes low-pass filtering (LPF) processing and reduction processing on image data to separate the input image data into a plurality of pieces of image data of different frequency bands. Then high band components of the plurality of pieces of image data corresponding to each frequency band are subjected to noise reduction according to edge information which is calculated based on low band components of the image data of each frequency band. Subsequently, the image data of high frequency bands, each of which having been subjected to the noise reduction, and the image data of low frequency bands is combined together.
A configuration of a common image processing unit which first separates input image data into a plurality of pieces of image data of different frequency bands and combines the plurality of pieces of image data of different frequency bands together later will be described in detail below with reference to FIG. 5.
Referring to FIG. 5, an image processing unit includes a division unit 510, a memory 520, a correction unit 530, and a combining unit 540. The division unit 510 divides input image data into a plurality of pieces of image data of different frequency bands. The correction unit 530 executes correction on the image data of each frequency band. The combining unit 540 combines the image-processed plurality of pieces of image data of different frequency bands together.
The division unit 510 outputs input image data Sin to the memory 520 as image data of a first frequency band. The division unit 510 then inputs the input image data Sin to a high-pass filter (HPF) 511. Further, the division unit 510 outputs image data generated by extracting high frequency components to the memory 520 as the image data of the first frequency band. The division unit 510 then inputs the input image data Sin to a low-pass filter 512 to execute smoothing on the image data Sin. Subsequently, the smoothed image data is input to a reduction circuit 513. Accordingly, image data of pixels less than those of the input image data Sin is generated. Further, the division unit 510 inputs the image data to a high-pass filter 514. Moreover, the division unit 510 outputs image data generated by extracting high frequency components to the memory 520 as image data of a second frequency band.
In addition, a low-pass filter 515 executes smoothing on the image data generated by the reduction circuit 513. The smoothed image data is then input to a reduction circuit 516. Accordingly, image data of pixels less than those of the image data generated by the reduction circuit 513 is generated. The image data is then output to the memory 520 as image data of a third frequency band. The frequency band is the highest in the first frequency band and becomes lower to the second and the third frequency bands in this ascending order.
In the division unit 510 illustrated in FIG. 5, the high-pass filter, the low-pass filter, and the reduction circuit constitute one band division circuit. In other words, the division unit 510 includes two band division circuits which are provided in a tree-like configuration.
The correction unit 530 includes three correction circuits 531 through 533 which correspond to the image data of the first through the third frequency bands, respectively. The correction circuits 531 through 533 executes various processing, such as noise reduction and edge information storage processing for generating a high definition image, or aberration correction, on the image data of each frequency band read from the memory 520.
The combining unit 540 inputs the image data of the third frequency band, which has been processed by the correction circuit 533, to an enlargement circuit 541 to generate image data of the same number of pixels as that of the image data of the second frequency band, which has been processed by the correction circuit 532. Then the generated image data is smoothed by a low-pass filter 542. An adding circuit 543 adds the image data of the second frequency band processed by the correction circuit 532 to the image data output from the low-pass filter 542. An enlargement circuit 544 generates image data of the same number of pixels as that of the image data of the first frequency band, which has been processed by the correction circuit 531, based on the image data output from the adding circuit 543. The generated image data is then smoothed by a low-pass filter 545. An adding circuit 546 adds the image data of the first frequency band processed by the correction circuit 531 to the image data output from the low-pass filter 545 to generate output image data Sout.
In the combining unit 540 illustrated in FIG. 5, the enlargement circuit, the low-pass filter, and the adding circuit constitute one band combining circuit. In other words, the combining unit 540 includes two band combining circuits which are provided in a tree-like configuration.
As described above, in image data acquired by an image sensor having primary color filters of the Bayer array, which includes color filters for a plurality of colors arranged in a mosaic-like arrangement, a pixel value corresponding to only one of the plurality of colors can be acquired from one pixel.
However, Japanese Patent Application Laid-Open No. 2008-015741 does not discuss any method for processing image data whose each pixel includes a signal corresponding to any color of the plurality of colors of the color filters.
In the case of a single plate image sensor with a primary color Bayer array, each pixel of the image sensor can have signals of all color components by executing reduction processing, which is implemented by downsampling executed when the image data is divided into a plurality of pieces of image data of different frequency bands, without interpolation on pixels having zero signals for each color. As illustrated in FIG. 6, to particularly describe a specific row of an image sensor having the Bayer array, an R signal color filter and a G signal color filter are recursively arranged in this order. A conventional method for reducing the number of pixels arranged in this manner to half the original number in the horizontal direction will be described below. To particularly describe the R signals of the specific row, the R signals which are originally arranged in the period of one R pixel for two pixels are reduced into one signal for one pixel by the reduction processing. Similarly, to specifically describe the G signals of the same row, the G signals which are originally arranged in the period of one G pixel for two pixels are reduced into one signal for one pixel by the reduction processing. However, during the processing for reducing the G signals, in order to align a barycentric position of the G signals with a barycentric position of the R signals, a value of the barycentric position is calculated by averaging pixel values of adjacent pixels. By executing reduction in the above-described manner, signals of all the color components of RGB primary colors can be generated at one pixel position without interpolation on pixels having zero signals for each color. In general, to generate signals of a plurality of colors for the same pixel is generally described as synchronization.
Because the image data of the second frequency band and the image data of the third frequency band have been synchronized by the reduction, the image data of the second frequency band and the image data of the third frequency band can be stored on the memory 520 in the synchronized state. If the synchronized image data of the second frequency band and the synchronized image data of the third frequency band is reversely converted back into image data of the Bayer array, information amount of the image data may decrease due to the reverse conversion. Thus, the accuracy of correction by the correction unit 530, which is executed by noise reduction or aberration correction, may degrade.
Suppose that the image data of the first frequency band may be subjected to the synchronization associated with the image data of the second frequency band and the image data of the third frequency band and then the synchronized image data is stored on the memory 520. However, in this case, the information amount of the image data of the first frequency band, whose information amount is the largest data, may become as three times as large. Therefore, it becomes necessary to increase the capacity of the memory 520 to an extremely large capacity.